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Abstract 

The intent of this study was to evaluate the Guide on the Side (GotS), an online learning tool 
developed by the University of Arizona Libraries, and a screencast tutorial for teaching 
information literacy and database searching skills. Ninety undergraduate students were 
randomly assigned into three groups: group 1 completed a GotS tutorial; group 2 viewed a 
screencast presenting identical content; and a control group. Each group completed an identical 
16-item post-test. An analysis of variance revealed statistically significant differences between 
the control group and both treatment groups; however, there was no statistical difference 
between treatment groups. Limitations of the study and future research areas are also discussed. 


70 





Mery, DeFrain, Kline & Suit, Evaluating the Effectiveness 

Introduction 

The ability to locate and use information 
effectively for a specific purpose is a main 
objective of infonnation literacy instruction. 
College students often proclaim that they 
possess the research skills to search and find 
information; yet when put to the test, 
librarians and instructors often find 
students’ infonnation literacy skills in 
discord to their boasting confidence 
(Oblinger, 2008). Students cannot be faulted 
since the ease of finding information online 
via search engines often gives a false sense 
of self-assurance. To complicate matters 
further, free and commercial information 
systems with varying designs continue to 
constantly evolve in the marketplace. 
Library databases vary greatly in their 
features and their content; students need 
direct instruction in the use of different 
databases in order to understand when and 
how to use them. 

With the ubiquity of Google, it is even more 
important that students understand the 
substantial differences between information 
sources found on the web and those found in 
library databases. Publications from the 
Project Information Literacy ongoing 
research initiative and many others have 
well-documented the problems students 
encounter when conducting their own 
research, from presuming that everything is 
available and therefore findable through 
web search engines, to discerning what are 
credible, scholarly, and/or relevant sources 
from the millions of results located through 
Internet searches (Head & Eisenberg, 2009; 
Head & Eisenberg, 2010). Based on studies 
such as these, the authors feel strongly that 
effective use of databases should continue to 
be an essential component of bibliographic 
instruction. 

In 2000, due to the growing prominence of 
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online courses and shrinking staff resources 
from economic challenges, the University of 
Arizona Libraries (UAL) began 
investigating the plausibility and scalability 
of using online learning as a means of 
transitioning away from the resource¬ 
intensive face-to-face model of information 
literacy instruction. Similar to many other 
academic libraries around the country at the 
time, UAL librarians began creating 
database demonstration videos using 
screencasting software. Although the videos 
were simple to create, there was concern 
that they lacked interactivity and any means 
of assessing learning, and were therefore not 
a viable alternative to classroom instruction 
(Suit, 2013). 

Seeking a tool that could replicate recursive 
classroom instruction while supporting the 
primary objective of teaching database 
searching skills, UAL librarians began 
developing a web-based learning tool to 
guide learners through navigating a live 
website or database by presenting a series of 
linear steps and activities on the left-hand 
side of the screen. The creators believed that 
the tutorial fonnat, at its core, would 
successfully employ active learning theory 
by allowing students to improve their 
researching skills in an authentic, real-time 
environment (Suit, Mery, Blakiston, & 
Kline, 2013). In addition, when compared to 
basic screencast videos, instruction 
librarians felt that these types of tutorials 
would more closely support the best 
practices widely recommended for 
classroom instruction: ensuring that the 
learning taking place is active/interactive 
(Blummer & Kritskaya, 2009; Dewland, 
1999; Mayer & Chandler, 2001; Oud, 
2009); providing students with clearly stated 
objectives (Blummer & Kritskaya, 2009; 
Dewland, 1999; Oud, 2009); teaching 
concepts as well as procedural knowledge 
(Dewland, 1999; McGuigan, 2001; Mestre, 
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2012); providing context specific feedback 
(Dewland, 1999; Oud, 2009); and clearly 
tying library instruction to class assignments 
(Dewland, 1999). 

The tool, now known as the Guide on the 
Side (GotS), has gone through a number of 
substantial iterations, most recently a 
considerable design update along with the 
creation of an easy to use WYSIWYG 
(What You See is What You Get) 
administrative interface to circumvent the 
need for web programming skills. Its 
subsequent release as an open source 
download in 2012 has garnered considerable 
positive national attention, and has also 
prompted numerous questions regarding 
both its effectiveness as an instructional 
tool, and guidance on how to best use it. 
While a growing number of publications 
have addressed the latter issue (Suit, Mery, 
Blakiston, & Kline, 2012; Suit, 2013; 
DeFrain, Mery, Suit, 2013), no empirical 
data had been gathered with regard to the 
former since a 2002 pilot study, which 
concluded that the tool was “a model for 
reaching large numbers of students” (Bracke 
& Dickstein, p. 330). That study evaluated 
the tool as a standalone 
instructional device by comparing it to face- 
to-face instruction. A genuine curiosity for 
evaluating the GotS, along with a true 
academic need to test the assumption that it 
is an excellent tool for online learning were 
therefore the impetuses for this study. 

Literature Review 

Screencasting software developments over 
the past decade rendered programs easier to 
use, less expensive, and nearly ubiquitous in 
library instruction. As more and more 
library instruction moves online, a great 
deal of database instruction is carried out 
through the use of tutorials created with 
screencasting software. A 2008 study on 
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online tutorial creation by libraries at one 
hundred randomly selected colleges and 
universities revealed that 40% of all 
tutorials created focus on database 
instruction (Yang, 2009). The study also 
found that librarians used screencasting 
software to create 33% of the tutorials they 
offered. Although a number of libraries are 
working to design screencast tutorials with 
interactivity that goes beyond the simple 
clicking of a forward or back button or 
usage of multiple choice questions (Betty, 
2008; Sherwill-Navarro & Layton, 2006), 
most libraries continue to rely on a more 
traditional and less interactive approach 
where students watch a narrated video 
demonstration to leam how to search a 
database. 

In a recent analysis of the literature, 
Stiwinter (2013) found that while “the 
importance of interactivity in an online 
tutorial was the most frequently mentioned 
trait” (p. 19), none of the library tutorial 
studies evaluated actually contained the 
desirable level of interactivity. This may be 
due in part to the fact that more interactive 
tutorials require a larger time commitment 
(Alyse, Ergood, Padron, & Reber, 2012; 
Xiao, Pietraszewski, & Goodwin, 2004; 
Sherwill-Navarro & Layton, 2006). After 
spending six months developing one video 
tutorial, Gravett (2010) concluded that the 
project was “significantly more time- 
consuming than expected” (p. 70), and 
expressed uncertainty over the project’s 
future. 

Even though screencast tutorials are 
ubiquitous in library instruction, there is 
little research of their effectiveness and even 
fewer empirical studies on screencasts 
(Lloyd & Robertson, 2012). The few 
studies that do exist about the effectiveness 
of screencast tutorials result in mixed 
findings. A study conducted by Mestre 


72 



Mery, DcFrain, Kline & Suit, Evaluating the Effectiveness 

(2012), which examined both student 
preferences for tutorial design as well as the 
efficacy of screencasts versus static 
webpages with screenshots, found that 16 of 
21 students preferred the static webpages 
with screenshots over screencasts. Mestre 
also found that students performed 
significantly better on post-tests after using 
the static webpage with screenshots than 
they did when using the screencast tutorial. 

In their study on the effectiveness of 
screencast tutorials to teach statistics to 53 
upper level psychology students, Lloyd and 
Robertson (2012) found the opposite to be 
true. They found that students who watched 
a screencast tutorial outperformed students 
who were given a text tutorial on two 
different sets of tests. It should be pointed 
out that these studies compare the 
effectiveness of screencast tutorials with 
even more passive forms of instruction, in 
both cases text-heavy formats. A recent 
study conducted by Sachs, Langan, 
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Leatherman, & Walters (2013) compared 
the information literacy outcomes of 
millennial undergraduate students that took 
either a “traditional” text-heavy tutorial or a 
more interactive, “millennial friendly” 
tutorial. In this study, the researchers found 
“very little difference in student learning 
outcomes connected to the two tutorials” (p. 
334). The researchers found that even 
though students’ performance was similar, 
they had a “strong overall preference” (p. 
334) for the “millennial friendly” tutorial. In 
examining these different studies, it 
becomes clear that tutorials that rely on text 
as well as those that use interactivity can 
successfully teach students information 
literacy skills. 

Purpose of Study 

The purpose of this study was to evaluate 
the effectiveness of the GotS tutorials and 
screencast tutorials in teaching 
undergraduate students online database 


Figure l—S creenshot of the Guide on the Side tutorial 
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searching skills. Two types of tutorials were 
used in this study: the GotS (Figure 1), and 
a screencast tutorial (Figure 2). Both 
tutorials focused on teaching students how 
to use the Academic Search Complete 
(ASC) database. The UAL already had a 
GotS for the database so only minor 
revisions to the existing tutorial were made 
for the study. Researchers then created an 
equivalent six-minute screencast tutorial 
using Adobe Captivate 5.5. Both tutorials 
are self-paced and centered around the task 
of locating articles for a specific research 
question, How do social networking sites 
such as Facebook affect romantic 
relationships? The main learning objectives 
for both tutorials are listed below: 

• Accessing Academic Search 
Complete 

• Identifying keywords 

• Using Boolean operators 

• Evaluating search results 

• Reading an article record 

• Accessing articles 

Figure 2—Screenshot of the s 
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• Citing articles 

• Distinguishing scholarly and 
popular articles 

• Locating scholarly articles 

• Retrieving articles that are not full- 
text 

The authors were also interested in gauging 
whether any inherent differences in 
information retention exist between the GotS 
and the screencast. It was anticipated that the 
GotS would prove to be a more effective 
instructional tool due to its interactive features 
and hands-on practice. Thus, the authors 
developed two research hypotheses: 

HI: Asynchronous online instruction 
is an effective means of teaching 
database searching skills. 

H2 : There will be a significant 
difference in post-test scores between 
students who complete the GotS 
tutorial and students who view the 
screencast tutorial. 

INCAST TUTORIAL 
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Methodology 

In addition to the two tutorials used in the 
study, researchers created a set of 16 
multiple choice test items based on these 
same learning outcomes (see Appendix 
1). In order to establish content validity, the 
items were shown to several librarians and 
revised according to their feedback. Each 
test item included four possibilities plus an 
“I don’t know” option so that students 
would not be forced to make a selection 
when they did not know the answer. The 
test items were loaded into WASSAIL, an 
open source web-based database from the 
University of Alberta Libraries that allows 
for the creation, management, and delivery 
of test items. 

Researchers designed a post-test-only 
control group study. Participants were 
randomly assigned to one of three groups: 
control, screencast, or GotS. The control 
group took the post-test; the screencast 
group watched the screencast tutorial 
followed by a post-test; and the GotS group 
completed the GotS tutorial followed by a 
post-test. 

Ninety undergraduate students were 
recruited for the study via an advertisement 
in the local student newspaper, flyers posted 
around campus, and an information table set 
up towards the entrance of the main library. 
Of all these recruitment methods, the table 
set up in the library resulted in securing the 
most participants. In order to participate in 
the study, students needed to be over 18 
years of age, currently enrolled as 
undergraduate students, and inexperienced 
with any type of library instruction at the 
University. Students were also asked 
whether they were familiar with ASC or 
other library databases. If students answered 
“yes” to one of these last two questions they 
were not eligible to participate in the study. 
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If students answered “no,” they were 
randomly assigned to one of the three 
groups. 

Once students were assigned to a group, 
they were placed in a room and given 
instructions on how to access the tutorials or 
the test. After students finished a tutorial, 
they were given the test online in the same 
room. Upon completing the test, students 
were given a $15.00 gift card for the 
University’s bookstore. 

Data Analysis 

Using version 20.0 of the Statistical 
Package for Social Sciences (SPSS) 
software, a one-way analysis of variance 
(ANOVA) was conducted on the students' 
post-test scores to detect any significant 
differences among the three groups of 
students. The significance level was set at a 
= .05. 

Results 

A total of 90 participants successfully 
completed the research study. The means 
and standard deviations recorded for each of 
the three groups are shown in Table 1. 
When analyzed using a one-way ANOVA, 
results indicated a significant difference on 
post-test scores among the three groups (F 
(2, 87) = 10.009, p < .001, q 2 = .187). 

Due to the significant F-value and equal 
sample sizes of the three groups, the post- 
hoc test Tukey's Honestly Significant 
Difference (HSD) was selected to determine 
which group means were significantly 
different from one another. Participants in 
the control group received significantly 
lower scores on the post-test, M= 8.17, 95% 
Cl [7.08,9.25] than those who viewed the 
screencast, M = 11.43, 95% Cl 

[10.35,12.52] or completed the GotS 
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tutorial, M = 10.77, 95% Cl [9.68,11.85]. 
As hypothesized, there was a significant 
difference in post-test scores among the 
control and the screencast groups (p < .001) 
and the control and the GotS groups (p 
= .003). However, the differences between 
the screencast and GotS groups failed to 
reach the significant level ip =.664), 
indicating that there was no difference in the 
effectiveness of the two types of instruction 
methods when measured with this 
questionnaire. 

Discussion 

It is clear from the analysis of test scores 
that both online instruction methods were 
effective, thereby confirming our first 
research hypothesis that database searching 
skills can be successfully taught online. 
These results also provide additional 
empirical evidence in the information 
literacy teaching field that database 
instruction can be successful when delivered 
asynchronously online. Furthennore, the 
control group’s low scores (M = 51.04%) 
indicate that undergraduate students who 
have not received library instruction 
generally lack these basic research skills. 
Thus, it can be asserted that dedicating time 
and resources to developing online learning 
tools is a worthwhile time investment for 
librarians. 

Surprisingly, the type of instruction received 
did not impact student performance on the 
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post-test. Using active learning theory as 
inspiration, the expected hypothesis posited 
the GotS as a more effective tool for 
delivering instruction than the screencast 
tutorial. Research regarding active learning 
theory suggests that students have a greater 
capacity for learning when they are actively 
engaged in the learning process (Prince, 
2004). The GotS tool was strategically 
developed over many years and iterations 
with active learning theory in mind (Suit, 
2013). As such, the authors believed that 
students who interacted with a database 
while learning to use it would retain more 
from the instruction than those who 
passively received information by merely 
watching a screencast. This study did not 
confirm this assumption, though several 
possible explanations may indicate why 
there were no observed differences between 
the groups, including the possibility that 
both instructional modes are effective. 

Limitations 

Creating two nearly identical instructional 
tools delivered in such different fonnats 
proved challenging. The study revealed that 
each format possesses strengths and 
weaknesses regarding various learning 
objectives and instructional methodologies. 
For example, upon reflection the GotS 
section on Boolean searching appeared 
confusing and overly detailed, particularly 
when compared to how this concept was 
handled in the screencast. Additionally, the 


Table 1—Number and percent of correct answers by online 

INSTRUCTION METHOD 



N 

Mean # Correct Answers 

Standard Deviation 

Control 

30 

8.17(51.04%) 

3.13 

Screencast 

30 

11.43 (71.46%) 

2.50 

Guide on the Side 

30 

10.77 (67.29%) 

3.28 
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GotS tutorial used in the study contained 
over 2,800 words and numerous searching 
exercises and knowledge check questions. 
While these interactive features are intended 
to assist with transfer and retention of skills, 
completing them required considerable time 
commitment from each participant. Where 
the screencast used in the study was only six 
minutes and 24 seconds in duration, it was 
noted that participants took anywhere from 
15 to 45 minutes to complete the GotS 
tutorial. The length of the GotS tutorial 
likely contributed to two outlier scores that 
were far below the average of all three 
groups. Removing these outliers from the 
data did not significantly affect the results. 
It is probable that these students felt rushed 
to complete the study and therefore did not 
take the quiz seriously. 

In the initial screening process, the authors 
learned that several students were non¬ 
native English speakers, potentially 
hindering their ability to understand the 
instruction or successfully complete the post 
-test. As the groups were randomly 
assigned, it is possible that one group 
contained more non-native English speakers 
than another, but because demographic 
information was not gathered, there is no 
way to correlate English proficiency and 
students’ post-test scores. 

The post-test developed for the study was 
also a limitation. Out of respect for 
participants’ time, the post-test was 
intentionally brief and contained only 16 
multiple choice questions. Having more test 
items would have been beneficial to gaining 
a deeper understanding of students’ abilities 
as it is difficult to determine if a student 
truly gained a skill based on one or two 
multiple choice items. There was one item 
where the control group outperfonned both 
treatment groups (item number 5) and one 
item where all groups fared equally (item 
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number 11). It is unknown what effect, if 
any, these particular items had on test 
scores. These types of issues indicate that 
some of the items on the post-test were 
poor. Unfortunately, the authors were not 
able to conduct any tests to measure the 
reliability of post-test items. Similarly, the 
post-test may have also suffered from 
validity issues. 

Future Research 

This study has opened up many 
opportunities for future research. The 
authors would like to continue to explore 
the effectiveness of the GotS by doing a 
similar study, but expanding data collection 
to incorporate participant characteristics 
such as major and class-standing. This 
information could be helpful in determining 
at what point in time to introduce specific 
information literacy concepts to students, 
and perhaps help the library in forming 
more strategic collaborations with 
disciplines needing an increase in research- 
related instructional efforts. In addition, the 
authors would like to measure the impact of 
different types of online tutorials on long¬ 
term acquisition of skills by testing students 
weeks or months after viewing a tutorial as 
opposed to immediately after. 

Student preferences regarding online 
delivery methodology and technological 
tool selection is also an important area 
needing further exploration. This 
experiment did not capture qualitative 
information or attempt to gauge the 
learners’ level of satisfaction with either of 
the tools, but feedback from users of GotS 
tutorials has been noticeably positive. Early 
on, this new instructional tool was well 
received by students, faculty, and librarians. 
Feedback from these groups shows that the 
tutorials can be fun, interactive, infonnative, 
and valuable. Where quantitative test scores 
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are an important part of evaluating 
instruction, understanding the help-seeking 
behaviors, motivating factors, and learner 
preferences of students is equally critical, 
particularly if librarians plan to continue to 
develop asynchronous instruction tools that 
are rarely assigned for-credit. 

Conclusion 

This study presents findings from an online 
tutorial research study that evaluated 
different methods of teaching database 
skills. When compared to the control group, 
students who completed different online 
tutorials showed significantly higher results 
on a post-test. Results also indicate that the 
type of online instruction students receive 
may not matter. That is, database instruction 
can successfully be taught online in a 
number of ways from static tutorials to 
highly interactive ones. 

The limitations addressed in this study 
indicate that several variables such as the 
length of the GotS, poor test items, and 
subject selection could have contributed to 
the outcomes. This study enabled useful 
improvements for the GotS tutorials 
including eliminating redundant text, 
simplifying instructions, and shortening its 
length. The authors have also revised test 
items and created new ones. These revisions 
will permit replicating the study in order to 
gain a better understanding of the pedagogy 
needed to create effective online database 
instruction. 

Endnote 

1. The authors were awarded a $2,000 
grant from the UAL Faculty 
Assembly in 2012 to help support 
the study; this money was used for 
the gift cards. 
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Appendix 

Post-Test Items (Correct answers are 
italicized) 

1. Which path would you follow to access 
Academic Search Complete from the UA 
Library Homepage? 

• Reference Resources —> Library Catalog 

• Search & Find —> Articles & Databases 

• Help —> Digital Collections 

• Services —> Document Delivery 

• I don't know 

2. Which statement best describes Academic 
Search Complete? 

• Academic Search Complete offers 
access to scholarly scientific articles 
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from the most popular disciplines on 
campus. 

• Academic Search Complete is a full-text 
peer-reviewed database that offers 
students access to the most widely used 
newspapers in the US and abroad. 

• Academic Search Complete is a 
database with access to academic papers 
including the complete works of popular 
professors and other scholars. 

• Academic Search Complete is a multi¬ 
disciplinary database with access to 
thousands of items from scholarly and 
popular resources. 

• I don't know 

3. Identify the best keywords for the 
following topic: Should federal courts in the 
United States permit the television coverage 
of trials? 

• United States, coverage, television 

• United States, federal, courts 

• Federal courts, trials, television 

• Television, pennit, federal 

• I don't know 

4. You type the following in Academic 
Search Complete: African Americans in the 
United States Supreme Court. How will 
Academic Search Complete treat this 
search? 

• It will automatically add the word AND 
in between each word 

• It will search for the exact phrase 

• It will treat the words as a title 

• It will look for articles that contain some 
but not all of these words 

• I don't know 

5. You are looking for articles that discuss 
how the government regulates school 
lunches. You ran the following search: 
government AND regulate and got over 
5100 results. What's the best way to 
decrease the number of results? 
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• Truncate the word regulat* 

• Use OR instead of AND in the search 

• Add another keyword with the AND 
connector 

• Enclose keywords from the search in 
quotes 

• I don't know 

6. If you search diet* AND atherosclerosis, 
which statement best represents the 
expected results: 

• You want the two terms to be searched 
as a phrase 

• You retrieve articles with the term 
dietary 

• You want the term diet to be searched 
first 

• You retrieve articles that contain 
synonyms for diet 

• I don’t know 

7. Which example best uses truncation? 

• psychologists* 

• automation* 

• neurolog * 

• computing* 

• I don't know 

8. You are searching for articles on teen 
pregnancy in the U.S. Select the search 
string that would retrieve the most relevant 
results. 

• (teen AND pregnancy) OR United 

States 

• teen* AND pregnancy AND United 

States 

• (teen OR pregnancy AND United 

States) 

• teen pregnancy in the United States 

• I don't know 

9. You search the tenns broadcast AND 
presidential race but are not happy with 
your results. By adding the connector OR 
such as in this search string: (broadcast OR 
television OR media) AND presidential race 
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- what would you expect to happen? 

• You will increase the number of results 

• You will decrease the number of results 

• You will accommodate for phrase 
searching 

• You will search different root word 
endings 

• I don't know 

10. In which type of periodical are you most 
likely to find scholarly articles? 

• Journals 

• Magazines 

• Newspapers 

• Catalogs 

• I don’t know 

11. Which statement best describes 
scholarly articles? 

• Scholarly articles are written by groups 
of peers at competing institutions. 

• Scholarly articles are written by 
professional journalists with excellent 
credentials. 

• Scholarly articles include a list of 
sources the author(s) used in the paper. 

• Scholarly articles are written to inform 
the general public about research results. 

• I don't know 

12. You are writing a paper on internet 
privacy and need to find some scholarly 
sources. Read each of the article records 
below by clicking on them and then choose 
the one that is scholarly. 

Link for article 1 
Link for article 2 
Link for article 3 
Link for article 4 

• Article 1 is scholarly 

• Article 2 is scholarly 

• Article 3 is scholarly 

• Article 4 is scholarly 

• I don't know 

13. You are writing a paper on online 
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dating. You have found quite a few good 
resources but still need a scholarly article. 
Click on the article below. 

Article Link 

Which type of an article is this? 

• This is a popular article 

• This is a scholarly article 

• This is an academic article 

• This is an editorial article 

• I don't know 

14. Look at the following article record by 
clicking on the link below and answer this 
question: What is the title of the journal in 
which this article is published? 

Article Record 

• Bonita Meyersfeld 

• Individual and Family Services 

• Marriage Law 

• Commonwealth Law Bulletin 

• I don’t know 

15. You find a great article in Academic 
Search Complete that will help you with 
your research question. What can you do to 
easily find similar articles? 

• Use one of the other library databases 

• Go to Ebsco’s home page to search for 
the title 

• Use the “Search the UA Library 
Catalog” link 

• Go through the bibliography or 
references list 

• I don't know 

16. What is the best way to evaluate how 
relevant an article you find in Academic 
Search Complete is to your research? 

• Look at the source 

• Look at the abstract 

• Look at the bibliography 

• Look at the subject terms 

• I don’t know 
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